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1.Purpose or Description: Please summarize how A.l is being used and for what purpose. 
1. AlDungeon.io is an implementation of the most advanced language transformer available, GPT-3, 
based around collaborative storytelling and gaming. 

We are using it as an experimental engine to test the use of deep learning Al in producing policy-related 
content, in surfacing future-related trends, and in automatically writing copy suitable for use in internal 
documents and published reports. 


2.Tool Name: What is the name of the tool or software that uses A.I? 
2. The name of the tool is AlDungeon.io. 


3. Algorithm Names: What algorithms does the tool use to implement the A.I solution? 

3. tis an implementation of the GPT-3 deep learning language transformer, optimized for collaborative 
storytelling and gaming. in addition, the collaborative storytelling component uses OpenlE5 for 
storytelling information extraction and the author's own algorithm AskBERT to extract information 
regarding locations, characters, and objects from provided text. The GPT-3 model has been fine tuned 
on a corpus of plot summaries collected from Wikipedia. More information on the algorithms 
implementation can be found in "Bringing Stories Alive: Generating Interactive Fiction Worlds", by 
Ammanabrolu, Cheung, Tu, Broniec, and Riedi, 2020. 


4.Scope: Internal - Not Citizen Focused, External - Impacts Citizens, Both. 
4, This algorithm is of internal scope, and does not directly impact government service delivery to 
citizens. 


5.Environment: Proof of Concept, Prototype, Development, In Production. 
5. This algorithm is being used for research purposes only. As such, it is at the proof of concept phase of 
maturity. 


6.Cost to Procure: Was the tool purchased, if so how much did it cost? 
6. This algorithm is being accessed on a subscription service. The cost of the service is $10/month. | 
believe the cost is in US dollars: we can check invoices to be sure. 


7.Employee Costs: Overall costs of time employees spend working on the АЛ solution. 

7. AlDungeon.io has incurred only the assessment, procurement, and oversight time in becoming 
available for use. | estimate the assessment time at two days of EC-6 time, and | estimate procurement 
time at 30 minutes of AS-4 and 15 minutes of EX-1 time including handshakes with senior staff for sign- 
off. In addition, | estimate that corporate services has incurred 45 minutes of time in updating working 
documents and providing oversight on this tool At current Treasury Board rates, the overall cost of time 
employees have spent working on the Al solution is $921.58. 


8.Professional Services Costs: Overall costs of professional services for the A.I solution. 
8. There was no professional services time incurred for this tool. 


9. Audits or Reports: Have any audits or reports been undertaken with respect to the АЛ 
solution (True/False)? 
9. No internal audits or reports have been produced with respect to this tool. 


10.Provide Copies of Reports - If Audits and or Reports exist with respect to the A. 
solution" 
10. There are no copies or reports to provide. 
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| 2020-02665" 
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following: 


HR Screening 


| 1. Purpose ог Description: 


x Please summarize how АЛ. is being used 
| and for what purpose. 


| This project aims to Identify the feasibility 
| of applying algorithmic tools to candidate 
| screening during staffing processes of 


| entry-level positions (high volume 
x processes, РМ-015 and А5-015 and such}. 


x Such a tool would be highly beneficial for 
| employees of HR departments who spend 
x a significant amount of time manually 

x screening applicants in such high volume 
x processes, 


x However, there are lot of considerations 
| in a project like this, such as fairness, 
| ethics and transparency. 


x The project is still very much at a 

x feasibility/ research phase and no real 
| deployment is planned for at least а 

| couple years (provided results are 

| positive and an approval is granted}. 


2.Too! Name: 


What is the 
name of the 
tool or 
Software 
that use A.L? 


HR Screening 
(Internal 
tool) 


3 Algorithm 
Names: 


What 
algorithms 
does the tool 
use to 


implement the 


АЛ. solution? 


Natural 
Language 
Processing 
(NLP), linear 
Regression, 
bag of words, 
Word 
embedding, 
neural 
networks 
transformer 


Internal- 
Not Crtizen 
Focused, 
External- 
impacts 
Citizens, 
Both 


internal- 
Not Citizen 
Focused 


5 Environment: 


Proof of 
Concept 
Prototype, 
Development, 
in Production. 


Proof of 
Conce pt 
(feasibility 
study) 


Cost to 
Procure: 


Was the too! 
purchase, if 
so how much 
did it cost? 


Tool 
not purchased 


| Overall 

| costs of 

| time 

| employees 
| spend 

| working on 
| the AL 

| Solution. 

| HRI 

| $80,000 (1 


| 2020-2021 
| fiscal year 


| 8.Professional 
Services Costs: 


| Overall costs of 
| professional 
| services for the 
| Ad. solution. 


| S. Audits or 
| Reports: 


| Have any 

| audits or 

| reports been 
| undertaken 

x with respect 
to the AL. 
solution 

x (True/False)? 


10.Provide Copies | 
of Reports: | 


H Audits and or 
Reports exist with | 
the respect to the | 
Aud, | 
Solution" 


See Lessons 
Learned 
Algorithms 
Screening- 
Lessons 
Learned.doc 


Alg Screening - 
Lessons Learned. do: 
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“Provide for 2020, 2021 a list of Al tools ESDC uses containing the following: 


1.Purpose or Description: Please summarize how A.l is being used and for what purpose. 
The purpose of this АЛ. based tool is to detect visual differences in documents submitted for Social 
Insurance Number (SIN) transactions that would suggest they have been manipulated prior to 
submission. 


2.Tool Name: What is the name of the tool or software that uses A.I? 
The tool uses packages in R and Python 


3. Algorithm Names: What algorithms does the tool use to implement the АЛ solution? 
The tool uses a convolutional neural network (VGG16 pretrained model) and autoencoders 


4.Scope: Internal - Not Citizen Focused, External - Impacts Citizens, Both. 
Both 


5.Environment: Proof of Concept, Prototype, Development, In Production. 
Proof of Concept 


6.Cost to Procure: Was the tool purchased, if so how much did it cost? 
No. 


7.Employee Costs: Overall costs of time employees spend working on the A. solution. 
To date 0.25 FTEs. Estimated going forward between 0.25 and 1 FTEs. 


8.Professional Services Costs: Overall costs of professional services for the A.I solution. 


$0 


9. Audits or Reports: Have any audits or reports been undertaken with respect to the АЛ 
solution (True/False)? 
False 


10.Provide Copies of Reports - If Audits and or Reports exist with respect to the АЛ 
solution" 
NA 


1.Purpose or Description: Please summarize how A. is being used and for what 
purpose. 
e Detecting confirmed fraud cases from ongoing investigations using investigator's 
text fields (notes). This relies on text mining (process text documents using 
human-defined patterns; regular expressions) 


2.Tool Name: What is the name of the tool or software that uses A.I? 
e Капа Python 


3. Algorithm Names: What algorithms does the tool use to implement the A. 
solution? 
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e Text mining tools: Stringr, regular expressions, text searching, filtering, 
extraction, etc 


4.Scope: Internal - Not Citizen Focused, External - Impacts Citizens, Both. 
e Both 


5.Environment: Proof of Concept, Prototype, Development, In Production. 
e POC/Prototype 


6.Cost to Procure: Was the tool purchased, if so how much did it cost? 
e $0 


7.Employee Costs: Overall costs of time employees spend working on the A. 
solution. 
e То date 0.25 FTEs. Estimated going forward between 0.25 and 1 FTEs. 


8.Professional Services Costs: Overall costs of professional services for the A. 
solution. 
e $0 


9.Audits or Reports: Have any audits or reports been undertaken with respect to 
the АЛ solution (True/False)? 
e FALSE 


10.Provide Copies of Reports - If Audits and or Reports exist with respect to the 
АЛ solution” 
e NA 
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From: rakesh.patry@ 
Sent: February 2, 2021 3:04 PM 
To: Pascale Archambault «Pascale. Archambault@ OSE 


SA 


Subject: ESDC $ identification oe the riek of forced labour 


Good afternoon Pascale, 
| hope this email finds you well. 


'm reaching out to kindly request that our file to procure the tool FRDM with reference # 
G9292-21-7353 submitted on January 26 by our ESDC Procurement team, be assigned 
a high priority. The file is presently in the hands of your group under Elizabeth Quenville 
and John Penhale. 


As you may know, ESDC is collaborating with the Canada Border Services Agency to 
implement the new Customs Tariff provision, which prohibits the importation of goods 
that are mined, manufactured or produced wholly or in part, by forced labour. Recent 
high-profile forced labour cases have attracted the attention of civil society, the media 
and federal ministers and our department is under mounting pressure as it is expected 
to actively monitor the risk of forced labour in global supply chains of Canadian 
Importers. 


Since these supply chains have also been under extreme pressure since the COVID-19 
pandemic started, this is a high priority not only for ESDC but also for GAC, CBSA and 
the Government of Canada. In parallel, we are working to address concerns raised by 
Minister Anand to Minister Tassi requesting ESDC's support to identify the risk of forced 
labour in supply chains of government suppliers, following the CBC Marketplace piece 
on PPE gloves from Malaysia. | would like to stress out that ће FRDM platform would 
be one of the key tools for ESDC's Labour Program to provide assistance to your 
department, should there be a concern with particular suppliers and their supply chain, 
and for our collaboration with the Canada Border Services Agency in their efforts to 
enforce the new Customs Tariff provision. 
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Please let us know how we can assist, in order to complete the procurement of the 
FRDM tool as soon as possible. 


Your valuable support is greatly appreciated. 
Best regards, 


Rakesh 


Rakesh Patry 

Directeur général | Director General 

Affaires internationales et intergouvernementales du travail | International and 
intergovernmental Labour Affairs 

Programme du travail | Labour Program 

Emploi et Développement social Canada | Employment and Social Development 
Canada 

Gouvernement du Canada | Government of Canada 

+(001) 613- in 7736 (cell) 

Rakesh.patrv@lab [ | 
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From: Patry, Rakesh R [NC] <rakesh.patry@labour-travail.gc.ca> 

Sent: September-26-19 9:50 AM 

To: Spielauer, Maria M [NC] «maria.spi ( : 

Cc: Barbier, Louise L [NC] «icuise.barbier?labour-travall.gc.ca»; Bouchard, Pierre P [NC] 
nierre.p.boucharc @labour-travall.gc.ca> 


Subject: RE: Follow up 


Hi Maria, 


Tony has confirmed that he is fine starting the procurement process now without waiting for the 
demo for the DM. 


He does want to know which procurement option IITB is selecting though. 
Thanks, 


Rakesh 


From: Spielauer, Maria M [NC] 

Sent: September 26, 2019 4:08 PM 

To: Patry, Rakesh R [NC] <rakesh.patry@labour-travail.gc.ca> 

Cc: Barbier, Louise L [NC] <louise.barbier@labour-travail.gc.ca>; Bouchard, Pierre P [NC] 
<pierre.p.bouchard@labour-travail.gc.ca>; Bazan, Jose Carlos JC [NC] <jose.bazan@labour- 
travail.gc.ca> 

Subject: RE: Follow up 


Hi Rakesh, 
Thank you very much for this information. 


The suggested approach to procurement would be under a pilot contract, which BRLA and IITB 
Data and Analytics Services (DAS) are presently exploring. 

In addition, BRLA gathered some valuable information from our federal partners (GAC and 
NRC), who have already passed TBS and PSPC scrutiny, and have provided it to IITB in 
support of this initiative (so we hope it works out). 

If this is not feasible, we will explore other options including a possible Request for Proposals 
(RFP). 


As a side note, today it was brought to our attention that DM Maheu confirmed with IITB (on 
September 17) the date for the second demo, which is planned for October 21 at 4:00 pm. 
ADMO is following up with DMO to agree on the venue and invitees. 


Please let us Know if you have any questions or need more information for Tony. 
Thanks, 
Maria 
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BRLA conducted research and identified 13 possible tools 


After analysis, the tool FRDM was identified as the most promising. 


Between August 31 and September 28, BRLA assessed FRDM's 
potential in assisting the division in the conduct of identifying the risk 
of forced labour in global supply chains affecting Canadian imports. 


Bilateral and Regional Labour Affairs, Nov 5, 2020 
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s.20(1)(b) 


[5 УУ 


БЕЧА БЕ RES ње 


v Technology based in San Francisco, California, specializing in the 
identification of the risk of forced labour in global supply chains 


v РЕКОМ manages massive amounts of data with the aid of Artificial 
Intelligence 


Y Data include transactional files such as shipping records and bills of 
lading, and adverse media 


Bilateral and Regional Labour Affairs, Nov 5, 2020 
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Bilateral and Regional Labour Affairs 
SAP confirmation on BRLA's Al expenditure 
March, 2021 


ESDC SAP confirmation on BRLA's AI ` costs 


Source: Business Services Unit 


| Date | SAP confirmation Amount 


Contribution towards 


31 march 2020 202167128 $100,000 total WEX pilot 
license costs 


Contribution towards 
WEX professional 
services for technical 
pilot implementation 


31 march 2020 9900016810 $34,000 


Budget Transfer 


Contribution towards 
WEX professional 
services for technical 
pilot implementation 


Date | SAP confirmation Amount 


5 November 2020 1000032209 $15,000 


FRDM subscription 


2/ March 2021 5100100249 $97,373.57 i 
for a one-year pilot 


** Artificial Intelligence technology 


Page 1/1 
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Background and Vision 


Proposed Solution 


Testing Watson with a Labour Program Query 


Operationalization of the Query 


Results and live demo 


Potential 


Proposed Next Steps 


The Labour Program Natural Language Processing Initiative, December 8, 2020 
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The Labour Program Natural Language Processing initiative, December 8, 2020 
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the ability to extract full value from Natural Language data, 


The Labour Program Natural Language Processing initiative, December 8, 2020 
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The Labour Program Natural Language Processing Initiative, December 8, 2020 
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The present analysis used automated intelligence, aided by existing 
manually produced and validated qualitative coding and results. 


The data sets were build and processed with the aid of ESDC 
infrastructure. 


Functional models can be generalized for similar needs from the 
same database for capacity building. 


These models could be adapted to other data sets, assuming there 
is knowledge of the related business rules in question. 
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Maternity Leave 


Transportation and 
Proportion from agreements 2% | Warehousing 


Total (Covered & Not) 


Covered 


Proportion Covered 


ample Narrative: 
From a sample of 2,300 unique collective agreement generations, 460 belonged to 
Transportation and Warehousing. From those, 147 contained paid maternity leave 
accounting for roughly 3296. 
This covered 193,966 employees or 66% of the industry's 293,318 employees. 
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puro 


Processing and analysis of higher volumes of data, with 
higher accuracy and smaller turnaround times 


Reduced risk of human error thanks to automation and easier 
detection due to consistent mistake patterns 


Less resources for higher-complexity tasking 
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Revamp the four pillars: what, why, how and when 


Build a data integrity and use case framework 
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> Watson Discovery Upgrade 


Infrastructure and Deployment options for the Labour Program 


Build a professional services and training plan 
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1.Purpose or Description: Please summarize how А.І |2.Tool Name: What is the name |3.Algorithm Names: What algorithms |4.Scope: Internal - Not Citizen 5.Environment: Proof of 6.Cost to Procure: Was 7.Employee Costs: Overall 8.Professional Services 9.Audits or Reports: Have any |10.Provide Copies of 
is being used and for what purpose. of the tool or software that uses | does the tool use to implement the  |Focused, External - Impacts Concept, Prototype, the tool purchased, if so  |costs of time employees spend Costs: Overall costs of audits or reports been Reports - If Audits and or 
A. solution? Citizens, Both. Development, In Production. |how much did it cost? working on the АЛ solution. professional services for the |undertaken with respect to the | Reports exist with respect 
АЛ solution. АЛ solution (True/False)? to the A.I solution" 


Approximately 10% of all Records of Employment received 
contain free text in the comment field of the form. These 
often contain relevant information that should 


Record of Employment (ROE)- 
comments 


Both in Production 


МЕР; APE Scripting Too! not purchased Approximate Salary Costs 
Record of Employment Dev Team = 
120,000 

Client Service Canada : 
Artificial Intelligence Centre o 
Expertise = 100,000 

Chief Data Office = 225,000 
Data and Analytics Services = 50,000 


Testing Services = 30,000 


ALIA is a search engine and content management system that |Artifical Intelligence Access Pilot Machine Learning Internal - Not Citizen Focused Proof of Concept, Prototype, Tool not purchased Approximate Salary Costs Proje 
will allow HR Officers in the Human Resources Services Branch |(ALIA) Development, team: $120K (ИТВ) 

to retrieve information efficiently and to replace the excel $65K (HRSC) 
spreadsheet they are currently using. ALIA use Elasticsearch, a 
smart search engine that has machine learning capability built 
in. 


Security specialis 


t S20k 


ct 


Other groups were implicated, such 
as IT Security, Innovation Fund, 
Accessibility, Business Relation 
Management, but those costs are not 
available for the moment. 


iService is an internal site at ESDC, where employees can iService Chatbot 
search for information. The iService Chatbot provides a 'eloped in-house using free, 
standard chatbot interface. The chatbot uses Al technology to rce software. 

help respond to user statements and questions. Total Approx. Costs = $100K 


NLP, RASA, ElasticSearch, API, Scripting 


Internal = Not Citizen Focused Tool not purchased. Was 


Approximate Salary Costs 
AIAI Team = $100K 


Al is being used to find insights from unstructured information IBM Watson Explorer Natural Language Processing, Deep Internal - No citizen or external The tool was used for a Proof of |The tool was purchased and Approximately $120,000 in IT salary 
such as documents and websites. The purpose was to evaluate Learning focused. Concept to see if insights could be |cost $250,000 to procure. were used to conduct the Proof of 
the tools ability to help analysts find extracted from concept. 

information from many desperate systems through a single MS Word documents, PDFs, and 

interface. Natural Websites. No decisions were 

Language Processing features were used to help find relevant made or automated. 

information for 

example positive and negative sentiments. 


Blue Prism is being assessed as an alternate product іл ESDC Blue Prism Form Recognizer - 14,385.00/yr unknown None 
to provide process automation with specialized capability in 

SAP automation including data entry and content migration. It : : 

assists in reducing the inventory / workload for agents | = 

processing Death Notifications and ISP 3017 reassessments by __ 

leveraging Blue Prism automation to partially or fully — 

complete the business process. 


Development language used in ESDC for enterprise grade R (Programming Language) Machine Learning, Linear regression, time  |Internal In Production No cost unknown unknown FALSE 
analytics and Business Intelligence Al application development series & statistical inference 
with supporting MS R Server. 


In Production unknown | I now : 
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SSPB Chief Data 
Office (CDO) 
input for A-2020- 
02665 "Provide 
for 2020, 2021 a 
list of Al tools 
ESDC uses 
containing the 
following: 


Handwritten text 
recognition for 
automated form 
processing 


Return T4 match 


Official Languages NLP 


2.Tool Name: 
What is the name 
of the tool or 
software that 
uses A.I? 


1.Purpose or Description: Please summarize how A. is being used and for what 
purpose. 


uth Employment 
ich апа Skills Strategy 
iallyinterested Project (Internal tool) 
preliminary 
n sections of the proposals. 
H, in Order to better 


Programs in ESD C still often rely on paper documents and require manual entry of the information from : MaRT-HTR (Internal 
these paper documents in our systems (i.e. to digitize the info). Some programs such as the pensions tool) 
program, which target older people who are often computer illiterate or have a strong preference 

toward paper-based forms. The pensions program is interested to see if it would be possible to 

automate (at least partially) this transcription with Optical Character Recognition (OCR) and/or 

Handwritten Text Recognition (HTR). 

While the field of Machine Learning already proven that these techniques are effective (at some 

degree) at solving these problems, there remain some practical obstacles that can complicate things — 

e.g., bad image quality and no reliable dataset to train or fine-tune models. 

This project aims to explore the feasibility to use Computer Vision tools on Optical Character 

Recognition (OCR) / Handwritten Text Recognition (HTR) problems in some ESDC processes (ex.: GIS). 


Return: 
(Intern. 


Canada Pension Plan (CPP) and Old Age Security (DAS) recipients are mailed As for their annual tax 
returns. About 250,000 14s are returned to sender due to various reasons such as address changes, 
which prompts an investigation into the client's file: Often; redundant work arises in that many clients 
have already contacted Service Canada to have their As reissued: This information is only stored in 
agent case notes, which previously needed to be manually read to close the file: 


Forthe past four years; the Chief Data Office (CDO) has supported this function using natural language 
processing (NEP) and machine learning (МЕ algorithms to predict, from agent case notes, whether TAS 
have been reissued: In addition; data from structured sources is also used to identify previously issued 
T4s:suchas client access of My Service Canada Account (MSCA/TISOF and recent client address changes 
in-QAS/CPP: master records. This year new approaches were taken to simplify the project's data 
processing pipeline by establishing automatic yearly recurring data extracts with HTB and by shifting the 
final result combination responsibilities to the data owners (the pension’s Workload Management and 
Process Automation Team}: This year; approximately 154,000 investigations were prevented (^4-6 FTEs} 
through the data analysis: 


This project aims to provide insight on the impact of language (French or English) on the merit of tools 
and techniques used in natural language processing (NLP). It aims to inform future decisions in NLP 
projects to allow data scientists to build better performing and less biaised solutions 


(Internal tool) 


Official Languages NLP Natural Language Both 


3.Algorithm 4.Scope: Internal 5.Environment: 
Names: What - Not Citizen Proof of 
algorithms does Focused, Concept, 

the tool use to External - Prototype, 
implement the Impacts Citizens, Development, In 
АЛ solution? Both. Production. 


Latent Dirichlet 
Allocation (EDA) 


In development 


NLP, neural networks, : Both 
BiLSTM, Transformers, 
CTC 


Proof of Concept, in 
development 


Proof of Concept, in 
Processing (NLP), development 
regression, neural 

networks, LSTMs, 

Transformers, BERT, 

Bag of words, GLOVE, 


fastText 
J in Production 


i-KMeans clustering, 
Label propagation 

b. For classification: 
i. Feed Forward 


Neural Network 


6.Cost to 
Procure: Was the 
tool purchased, 
if so how much 
did it cost? 


Tool not r 


Tool not purchased 


ool net purchased 


Tool not purchased 


8.Professional 
Services Costs: 
Overall costs of 


7.Employee 
Costs: Overall 
costs of time 


employees professional 
spend working services for the 
on the А. АЛ solution. 
solution. 


CDO - $72,000 (0.8 


None 
FTE) for 2020-2021 
fiscal year 


CDO- $9,000 (0.1 None 
for 2020-2021 fiscal 
year 


CDO - $18,000 (0.2 
FTE) for 2020-2021 
fiscal year 


None 


9.Audits or 10.Provide 
Reports: Have Copies of 
any audits or Reports - If 


reports been Audits and or 
undertaken with Reports exist 
respect to the A.I with respect to 
solution the A.I solution” 
(True/False)? 


FALSE None 


see Lessons Learned = 
Returned 14 - Lessons 
Learned:doc 


FALSE None 
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HR Screening This project aims to identify the feasibility of applying algorithmic tools to candidate screening during HR Screening (Internal NLP, linear regression, Internal - Not Citizen Proof of Concept Tool not purchased 1- CDO - $135,000 (1.5: None TRUE see Lessons Learned 


staffing processes of entry-level positions (high volume processes, РМ-015 and А5-015 and such). Such a tool) 
tool would be highly beneficial for employees of HR departments who spend a significant amount of 

time manually screening applicants in such high volume processes. However there are a lot of 

considerations in a project like this, such as fairness, ethics and transparency. The project is still very 


much at a feasability / research phase and no real deployment is planned for at least a couple years 
(provided results are positive and an approval is granted). 


bag of words, word Focused (feasibility study) FTE) 2- HRSB - Alg Screening - 
embeddings, neural $80,000 - Total Lessons Learned.doc 
networks, $215,000 for 2020- 

transformers 2021 fiscal year 


ional service desk 
(NSD) Ticket Triage 
(Internal tool) 


Natural Language Internal Not Citizen Proof of concept; іп: Tool not purchased. 
Processing (МЕР), Focused development TE) 
linear regression, jt ,000 tot 
neural networks fiscal ye 
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Chief Data Office 


Learning from Organizational Successes and Failures 


Name: _____| Returned T4 


Prepared Ву: Patrick Demers and Alexandre Parent 


Description: Canada Pension Plan (CPP) and Old Age Security (OAS) recipients receive 
annually a T4 form containing the benefits received that they need to report on their tax 
return. Many Т45 are returned to Service Canada due to address changes or other reasons 
and must be investigated. Each year, large numbers of clients follow up with Service Canada 
to request a duplicate T4, which is immediately reissued. However, the reissuance is only 
captured in call centre notes and a resource intensive manual investigation of every single 
individual returned T4 has still to performed to identify clients for which the T4 has already 
been reissued. 

This project uses machine learning, natural language processing and data linkage to 


proactively look at call centre agent's notes (unstructured data) in order to identify T4s 


already reissued and avoid duplication of work. Other linkages are done to remove further 
the manual workload – for example the MSCA data that tells if a client has accessed theirs 
T4s online. 


Rationale 


Returned T4 work-items are created because Service Canada has the obligation to get 
clients their T4 
When clients are put into pay state, they must keep us informed of their current contact 
information - so is the client fulfilling their obligation 
Off a mail item is returned, it would mean that the clients whereabouts are put into 
question (along with their eligibility for benefits) 
Note: This year is the 4'^ iteration of the project. 


Contributors to Lessons Learned Report: [Insert a bulleted list of the names of individuals 
that contributed to the lessons learned report – e.g. attended lessons learned meetings, one- 
off feedback. | 

Patrick Demers (CDO) 

Alexandre Parent (CDO) 

James Shin (CDO - data extract) 

Amielle Parisien (Pensions Workload Management & Process Automation, Program Advisor) - 

Business client 

Gwen Kinakin (Pensions Workload Management & Process Automation, Program Advisor) - 

Business client 


Page 1 of 3 


1 1 information is disclosed under the Access to information Act 
* Les renseignements sont divulgués en vertu de la Loi sur 
l'accés à l'information 


000090 


s.16(2) 


1 1 information is disclosed under the Access to information Act 
* Les renseignements sont divulgués en vertu de la Loi sur 
l'accés à l'information 


UNCLASSIFIED 


Sherry St-Onge (Pensions Business Solutions, Manager) 
lan Marshall (Pensions Business Solutions, Program Advisor) 
Valérie Paré-Eberley (Pensions Business Solutions, Director) 


Mazin Alkarkhi (IITB, MSCA) 


Successes: [Insert a bulleted list of what 
worked well in the accomplishment of the 
pilot/project/activity/deliverable.] 

For this year: 


Worked with multiple teams and 
stakeholders across ESDC including 
Pensions Workload Management & 
Process Automation, Pensions Business 
Solutions and IITB (My Service Canada 
Account) 

Project has been moved to version 
control system (internal gitlab) to track 
version code, track project issues and to 
make the code more accessible for other 
team members 

Significant amount of documentation has 
been added to the repository and will 
facilitate the integration of the project for 
new collaborators 

Reduced the processing workload by 
setting up reoccurring yearly data extracts 
of sins to be delivered directly to the 
client, instead of having to be merged by 
the CDO. 

Closed record number of work items 154k 
(out of 315k) about 4-6 FTEs 


Areas for Improvement: [Insert a bulleted 
list of what did not work well in the 
accomplishment of the pilot/project/activity/ 
deliverable.] 

e We did not have enough time to 
implement a Machine Learning Model 
Tracking system (MLflow) to 

o Record and query experiments: 
code, data, config, and results 
o Help ensure reproducibility 


Best Practices to Share [From the above-noted successes, list those 1-3 items that stand out 
as best practices which should be shared within the division.] 

Project has been moved to version control system (internal gitlab) to track version code, 
track project issues and to make the code more accessible for other team members 
Sample of predictions have been validated by the business experts 

Having clear and easy to follow documentation is important for when it is time to repeat 
the exercise and/or integrate new collaborators 


Next Steps for Improvement [List the next steps that will be undertaken with respect to 
identified areas for improvement. ] 
Brings the data on a central repo a.k.a. datalake — right now it’s : 
Implement a Machine Learning Model Tracking (MLflow) system 
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o This would help us to increase the model's performance in the next iterations of 
the projects 
e Use the OBDA to automate entire pipeline of receiving data from source, transformation, 
modeling & prediction, and results delivery to client. 


Other Comments [Insert any additional comments in this space] or Next Steps [Insert next 
steps or action items, if applicable] 

e We have been working with Pension Workload Management & Process Automation 
team for several years now on multiple projects, this is key stakeholder that plays an 
important role in service delivery operations for the Pension programs; maintaining 
relationships like these are key to help further transform the department towards 
doing more with data. 

e This is the 4" iteration of the project, and so we have ironed out many of the 
problems, that is why the list of "Areas for Improvement” is so short. 

e This project would make a great candidate for the oracle big data appliance (OBDA) 
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Chief Data Office 


Learning from Organizational Successes and Failures 


Name: Algorithmic screening 


Prepared By: Julien-Charles Lévesque and Jeff Hughes 


Description: This project aimed to identify the feasibility of applying algorithmic tools to 
candidate screening during staffing processes of entry-level positions (high volume 
processes, РМ-015 and А5-015 and such). Such a tool would be highly beneficial for 
employees and managers who spend a significant proportion of their time manually 
screening applicants in high volume processes. One of the goals of this project was also to 
identify existing biases in staffing processes. 


Note: This was the second phase of the project. 
Contributors to Lessons Learned Report: [Insert a bulleted list of the names of individuals 
that contributed to the lessons learned report — e.g. attended lessons learned meetings, one- 
off feedback. | 

Julien-Charles Lévesque (CDO) 

Jeffrey Hughes (CDO) 

Stephanie Sajous (HRBI) 

John Kehoe (HRBI) 

Danielle Blais (PSC OoR) 


Successes: [Insert a bulleted list of what Areas for Improvement: [Insert a bulleted 
worked well in the accomplishment of the list of what did not work well in the 
pilot/project/activity/deliverable.] accomplishment of the pilot/project/activity/ 
e Worked with multiple teams and deliverable.] 
stakeholders across ESDC including HRBI, | e Screening board reports came in multiple 
HR Ops, Legal services, Privacy. Also formats, which made assembling a 
worked with external partners such as the dataset for analysis and Al testing a lot 
Office of Renewal (OoR) at PSC. harder. 
Multiple reports and presentations have There seems to be no consensus on how 
been given to the partners, providing to perform screening for a specific set of 
insights about the way screening is skills. The level of requirements for the 
conducted across regions, as well as the same essential qualification criterion 
challenges of algorithmic screening. varies greatly across groups and regions 
e Analysis of historical screening data was in Canada. This makes it hard to develop 
not able to identify significant biases in and apply an artificial intelligence 
screening with regards to self-declared solution because there is no single right 
group membership (women, Aboriginal 
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peoples, persons with disabilities and answer for the screening of a given 

members of visible minorities). candidate. 
Some preliminary work into getting 
human raters to rescreen the same 
candidates multiple times showed high 
disagreement in screening decisions’. In 
other words, people often came to 
different conclusions for the same 
candidate. This also has implications for 
algorithm development. 
There are many ethical considerations in 
the application of algorithms for 
screening. One question that remains 
unanswered is whether 


Best Practices to Share [From the above-noted successes, list those 1-3 items that stand out 

as best practices which should be shared within the division.] 

e Coding practices. Code review, merge requests, and unit testing were used for major 
features in the model & analysis code. Highly recommended for other projects in the CDO 
where software development is required. 

Experiment tracking software (ex. MLFlow) was instrumental in exploring and comparing 
the performance of models on a variety of settings (e.g., hold-one-out process testing). 
Would recommend adopting such solutions in future AI projects. 


Next Steps for Improvement [List the next steps that will be undertaken with respect to 

identified areas for improvement.] 

e СОО recommendation: focus on standardizing and centralizing data collection for staffing. 
Perform in-depth analytics of said data and identify questions with experts from 
Workforce Management. 

At the moment, CDO's recommendation for algorithmic screening 


Al is better suited for problems where there is an obvious right answer 
for a given query. 


Other Comments [Insert any additional comments in this space] or Next Steps [Insert next 
steps or action items, if applicable] 
Decisions made: 


! This should be taken with reservations because the study had some methodological flaws (did not completely 


adhere to the standard procedure for screening). 
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e The ESDC Workforce team agrees with the recommendation of taking a step back in order 
to put the foundational pieces in place. They would like to reflect further on what they 
want to track, a central place to hold our data, and to identify future actions. 


e The PSC agrees with the recommendation as they believe that the purpose of screening is 


not only volume reduction, but also quality of hiring. As this project could go in many 
directions, they are interested in knowing what the priority in the future should be. 
Next steps: 
e First step would be to gather the data. Second step would be to find the questions we 
want answered and then attempt to answer them with the data. 


e Keep PSC іп the loop in order for them to be able to support as needed. 
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Algorithmic Ir 


Name of Respondent 


Marc-André Chicoine 


Job Title 


Senior Program Advisor - A&CS 
Department 


Branch 


TISMB 
Project Title 


ROE Comments Assessment using Artificial intelligence 


Project Phase 


Implementation 
[ Points: 0 | 
Please provide a project description: 


The Chief Data Office (CDO), Employment Insurance (El) Benefits Delivery Services (BDS) and 
Innovation, Information and Technology Branch (IITB) are developing and implementing an 
artificial intelligence (Al)/machine learning solution that will interpret and assess free text 
comments captured by employers when records of employment (ROE) are issued. Based on very 
comments, predict a different Reason for Separation [RFS]). The actions taken following an AI 
predictions will be saved within the internal interpretative notes used in the process of 
calculating El claims. The original document (Record of Employment [ROE]) will not be modified. 


What is motivating your team to introduce automation into this decision-making process? 
(Check all that apply) 


Lower transaction costs of an existing program 
Use innovative approaches 
Please check which of the following capabilities apply to your system. 


Text and speech analysis: Analyzing large data sets to recognize, process, and tag text, speech, 
voice, and make recommendations based on the tagging 

Process optimization and workflow automation: Analyzing large data sets to identify and 
anomalies, cluster patterns, predict outcomes or ways to optimize; and automate specific 
workflows 

Current Score: 41 

Raw impact Score: 41 

Mitigation Score: 30 
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Requirements Specific to Impact Level: 2 

Peer Review 

At least one of:Qualified expert from a federal, provincial territorial or municipal government 
from a relevant non - governmental organizationContracted third - party vendor with a related 


specializationPublishing specifications of the Automated Decision System in a peer - reviewed 
journalA data and automation advisory board specified by Treasury Board Secretariat 


Notice 
Plain language notice posted on the program or service website. 


Human-in-the-loop for decisions 


Decisions may be rendered without direct human involvement. 


xplanation Requirement 
In addition to any applicable legislative requirement, ensuring that a meaningful explanation is 


provided upon request for any decision that resulted in the denial of a benefit, a service, or other 
regulatory action. 


Testing 
Before going into production, develop the appropriate processes to ensure that training data is 
tested for unintended data biases and other factors that may unfairly impact the 


outcomes.Ensure that data being used by the Automated Decision System is routinely tested to 
ensure that it is still relevant, accurate, and up-to-date. 


Monitoring 
Monitor the outcomes of Automated Decision Systems on an ongoing basis to safeguard against 


unintentional outcomes and to ensure compliance with institutional and program legislation, as 
well as this Directive. 


Training 

Documentation on the design and functionality of the system. 
Contingency Planning 

None 

Approval for the system to operate 
None 


Link to the Directive on Automated Decision-Makina Impact Level Reauirements 
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The following internal stakeholders have been consulted: 


A documented process is currently in place to test datasets against biases and other unexpected 
outcomes. 


Accountabilities for the design, development, maintenance, and improvements for the system 
have been assigned. 


A process has been developed to manage any risks of having outdated or unreliable data that 
could be used in the system. 


This information is now publicly available. 

The system records all the recommendations or decisions made by the system. 
All key decision points are identifiable in the audit trail. 

All key decisions points are linked to the relevant legislation, policy or procedure. 


A change log has been developed to detail all of the changes made to the model and to the 
system. 


The system's audit trail can be used to help generate a notification of the decision (including a 
statement of reasons or other notifications) where required. 


The audit trail identifies which version of the system was used for each decision. 
The system's audit trail shows who the authorized decision-makers are. 
There is a process in place to grant, monitor, and revoke access permission to the system. 


There is a mechanism 


to capture feedback by users of the systems. 
The system enables human override of system decisions. 
There is a process in place to log instances when overrides were performed. 


The system's audit trail includes change control processes to record modifications to the 
systems operation or performance. 


Questions and Answers 


frequent litigation? 
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Yes | 
[ Points: +3] 
Are clients in this line of business particularly vulnerable? 


No 
| Points: +0 | 


Are stakes of the decisions very high? 


No 
[ Points: +0 | 
Will this project have major impacts on staff, either in terms of their numbers or their roles? 


No 
| Points: +0 | 
Will you require new policy authority for this project? 


No 
| Points: +0 | 


No 
[ Points: +0 | 
The algorithmic process will be difficult to interpret or to explain 


Yes 
[ Points: +3 | 
Does the decision pertain to any of the categories below (check all that apply): 


Social assistance (ei, disability claims, etc) 
Points: +1 | 
Will the system only be used to assist a decision-maker? 


No 
[ Points: +0 | 


Will the system be replacing a decision that would otherwise be made by a human? 


Yes 
[ Points: +3 | 
Will the system be replacing human decisions that require judgement or discretion? 


Yes 
| Points: +4 | 
Please describe the decision(s) that will be automated 


based on current procedures used by agents. Due to the simplicity of these decisions, the A! will 
only replace a percentage of these decisions.When the Al interpretation is used in a decision, it 
will replace a decision currently made by a human that requires minimal judgement and/or 
discretion. However, a portion of the decision (cant be assessed by Al or more complex 
decisions) will continue to be made by humans. 


Is the system used by a different part of the organization than the ones who developed it? 
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Yes 
[ Points: +4 | 
Are the impacts resulting from the decision reversible? 


Reversible 
[ Points: +1 | 
How long will impacts from the decision last? 


Some impacts may last a matter of months, but some lingering impacts may last longer 
| Points: +2 | 
Please describe why the impacts resulting from the decision are as per selected option above. 


Information received on a ROE can normally be used on a El claim for a period of one (1) year, 
excluding certain exceptions where the period is extended and possible appeals. 


The impacts that the decision will have on the rights or freedoms of individuals will likely be: 


Little to no impact 
[ Points: +1 | 
Please describe why the impacts resulting from the decision are (as per selected option above). 


As the prediction made by the Al is used in support of a larger eligibility decisions, and is 
reversible, the impacts resulting from the decisions have little to no impact on the rights or 
freedom of the individual applying for El benefits. 


The impacts that the decision will have on the health and well-being of individuals will likely be: 


E e eae 
[ Points: +1] 


Please describe why the impacts resulting from the decision are (as per selected option above) 


The А! decision has no relation, hence no impacts, to the health and well-being of individuals 
The impacts that the decision will have on the economic interests of individuals will likely be: 


Please describe why the impacts resulting from the decision are (as per selected option above) 


As it is completely reversible, the AI decision may only result in minor impacts to the speed at 
which the claim can be processed if additional verification is required on a specific decision 
made. In the vast majority of cases, the processing speed will be increased. 


The impacts that the decision will have on the ongoing sustainability of an environmental 
ecosystem, will likely be: 


Little to no impact 
[ Points: +1 | 


Please describe why the impacts resulting from the decision are (as per selected option above) 


The decision will have very little impacts, if any, on the sustainability of an environmental 
ecosystem 
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Will the Automated Decision System use personal information as input data? 


Yes 
| Points: +4 | 
What is the highest security classification of the input data used by the system? (Select one) 


Protected B / Protected C 
[ Points: +3 | 
Who controls the data? 


Federal government 
[ Points: +1 | 
Will the system use data from multiple different sources? 


No 

[ Points: +0 | 

Will the system require input data from an Internet- or telephony-connected device? (e.g. 
Internet of Things, sensor) 


NO 
[ Points: +0] 
Will the system interface with other IT systems? 


Yes 


[ Points: +4 | 
Who collected the data used for training the system? 


Your institution 
| Points: +1 | 
Who collected the input data used by the system? 


Your institution 

| Points: +1 | 

Will the system require the analysis of unstructured data to render a recommendation or a 
decision? 


Yes 
[ Points: 0 | 
What types of unstructured data? (Check all that apply) 


Audio and text files 
[ Points: +2 | 


Mitigation Questions and Answers 


Internal Stakeholders (Strategic policy and planning, Data Governance, Program Policy, etc.) 


Yes 
[ Points: +1 | 
External Stakeholders (Civil Society, Academia, Industry, etc.) 


No 

[ Points: +0 | 

Do you have documented processes in place to test datasets against biases and other 
unexpected outcomes? This could include experience in applying frameworks, methods, 
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guidelines or other assessment tools. 


Yes 
| Points: +2 | 
Is this information publicly available? 


No 

Points: 40 

Have you developed a process to document how data quality issues were resolved during the 
design process? 


No 
[ Points: +0 | 
Is this information publicly available? 


No 
[ Points: +0 | 


Have you undertaken a Gender Based Analysis Plus of the data? 


No 
[ Points: +0] 
is this information publicly available? 


No 

[ Points: +0 | 

Have you assigned accountability in your institution for the design, development, maintenance, 
and improvement of the system? 


Yes 

[ Points: +2 | 

Do you have a documented process to manage the risk that outdated or unreliable data is used 
to make an automated decision? 


Yes 
| Points: +2 | 


is this information publicly available? 


Yes 
| Points: +1 | 
is the data used for this system posted on the Open Government Portal? 


No 
[ Points: +0 | 
Does the audit trail identify the authority or delegated authority identified in legislation? 


No 
[ Points: +0 | 


Does the system provide an audit trail that records all the recommendations or decisions made 
by the system? 


Yes | 
| Points: +2 | 


Are all key decision points identifiable in audit trail? 
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Yes 

| Points: +2 | 

Are all key decision points within the automated system's logic linked to the relevant legislation, 
policy or procedures? 


Yes | 


Do you maintain a current and up to date log detailing all of the changes made to the model and 
the system? 


Yes | 
[ Points: +2] 
Does the systems audit trail indicate all of decision points made by the system? 


Yes 

Points: +1 | 

Can the audit trail generated by the system be used to help generate a notification of the 
decision (including a statement of reasons or other notifications) where required? 


Yes 

[ Points: +1] 

Does the audit trail identify precisely which version of the system was used for each decision it 
supports? 


Yes | 
| Points: +2 | 
Does the audit trail show who an authorized decision-maker is? 


Yes 
Points: +1 ] 
Is the system able to produce reasons for its decisions or recommendations when required? 


NO 
[ Points: +0 ] 


Points: +1 ] 


Yes 
[ Points: +1 | 
is there a recourse process established for clients that wish to challenge the decision? 


Yes 
[ Points: +2 | 


Does the system enable human override of system decisions? 


Yes 
| Points: +2 | 
is there a process in place to log the instances when overrides were performed? 


Yes 
[ Points: +1 | 
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Does the systems audit trail include change control processes to record modifications to the 
system's operation or performance? 


Yes 
[ Points: +2 | 


Have you prepared a concept case to the Government of Canada Enterprise Architecture Review 
Board? 


No | 
| Points: +0 | 
Have you completed a Privacy Impact Assessment or revised an existing one? 


Yes 
[ Points: +1] 
Does your system reflect Privacy by Design principles? 


No 
Points: +0 ] 
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Presented by 
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ITS TIME TO DO MORE WITH DATA 
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О. Who We Are 


° Employment and Social Development Canada 


— Mandated fo provide services and benefits to Canadians 
throughout entire lifesoan 


• Chief Data Office 


— Mission is fo empower the innovative, secure, and ethical 
use of data and analytics 


— Specializations in data strategy, intelligence, governance, 
policy, dissemination, and science 
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О. Who We Are 


« Data Science Team 


— Goal is fo use data to improve and enhance 
services provided to Canadians 


— Achieved by solving real world problems using advanced 
analytical techniques 
e Machine Learning/Al 
e Natural Language Processing 
• Computer Vision 
° Geospatial Analysis 
• Data Visualization 
« Software Development 
• IT Management 
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1. Business Context 


Mandated to look into possible projects with Service 
Canada 


° Opportunity to help Canadians on multiple 
fronts through: 
— Increasing efficiency of internal processes 
— Automation of tasks directly impacting service delivery 


лч ge 5 D 2 * 3 AN 8 , у 
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1. Business Context 
• ROE form targetted – Comment Field 


— ~11% of ROE coming in contain text in the comment field 
(~ 1.2 million ROEs with comments in 2019) 


— ROE Comments Work Items created for agents to review 
тле content 


— Example of comments: 
* "Mise à pied" 
o "Does not have the necessary skills for the position" 
o "Feu de forêt du 17 juin au 30 Juin" 
° “Relocating to Alberta" 
• "Conjointe infectée à la COVID-19" 
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1. Business Context 


Reason for Separation Frequencies 


180 


O. 
О 


140 
120 


Thousands 
со © 
O © 


N + © 
O O O O 


* Sub-categories of their respective Reason for Separation 
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2. ROE Processing Steps 


| by agent 


abase 
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2. ROE Processing Steps 


Al Model removes 
comment 
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3. ROEC Model - How does Ії work? 


° Text Classification 
— Categorize unstructured text into discrete groups 
e [ext Preprocessing and Tokenization/Vectorization 


— Preprocess text: remove punctuation, tag numbers, decapitalize 


— Tokenize and Vectorize: split text into smaller tokens, create numerical 
representations of the text That the machine learning algorithm can read 


Raw text Preprocessed text Vectorized text 
Mise a pied. ~ | mise a pied к. 14776, 93205, 65937, 


S.V.P contactez | [ > |svp contactez > | 04352, 88821, 41860, ..., 
123-456-7890 d «phone» d 31594, 49324, 54406] 
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3. ROEC Model - How does it work? 


° Trying to replicate actions of Service Canada Agents reviewing 
Comment Work Items 


e Neural Network 
— Training: 
• Learns patterns between input text and output label 
— Predicting: 
* Leverages learned patterns to predict on new data 


Input Machine Readable Format Output 


Label Confidence 


Free Text Label Word Vector Label 


Shortage of 
Work 


Mise à pied ен ДЕ » > | [03243, 59409 ... 43865] | Shortage of 


Machine Learning 


. / Моде! 


000115 


1 1 information is disclosed under the Access to information Act 
* Les renseignements sont divulgués en vertu de la Lol sur 
l'accés à l'information 


3. ROEC Model - How does it work? 


simplified Model Flow Chart 
EX e Developed in close 


| POE | у : š 
ш collaboration with business 
experts 


5% 95% 


Machine 


Regex Pattern | 
Learning Model 


Matching 


e Classifying between 10 
different categories 


( Prediction ^ | 
Ne. ft 
рр Í 

Business Rules | | 

55 * Considers the fact that 

E mem | depending on reason for 


Dates // Dollar | ° š 
separation, ассертеа risks 
NE and error costs will vary 


| Updated x 
ROE form 


eee, 
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3. ROEC Model - How does it work? 


Threshold Selection 


10076 


90% E 


2222222222 


— _ — 
70% —— 
60% ———— 
ЗО: — 
40% —— 
30% 
20% 
10% 


RS 


| 


8 


Least Restrictive Scenario Middle Ground Scenario Most Restrictive Scenario 
(lower thresholds) (higher thresholds 


* 96 of comments actioned 


Precision 


000117 


1 || information is disclosed under the Access to Information Act 
* Les renseignements sont divulgués en vertu de la Loi sur 
l'accés à l'information 


3. ROEC Model - Examples 


— Таван шш NE 


K-Other  |shortage of work 91.1% 
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4. Productionisation 


Made possible through the collaboration with several groups: 
business stakeholders, Al CoE, DAS (118), Privacy, Legal, etc. 


First Al Model fully integrated in an IT suite at ESDC. 


Algorithmic Impact Assessment to ensure compliance with TBS 
directives 


Solution in production since June 20th 2020 


000119 


5. Results 


° Current Performance 


— ~97% precision on last sample* (~2k comments) verified by a 
business expert 


— Since the solution is in production, 56% of ROEs with 
comments have been actioned by the model 


• [o Date 
— ~/30k ROEs have been automatically updated by the model 


— Estimated that ~170k updated ROEs were involved in El claims 
° ~170k ROE Comments Work Items eliminated 


*Sample from November 2020 
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5. Results 


• Сома Backlog 


— In August 2020, over 500k backlogged comments 
were passed through the model 


• ~58% of ROEs with comments were actioned (~285k 
ROEs with comments were removed from the backlog} 


• ~98% precision on sample of the data 
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6. Current Work and Next Phase 


° Currently closing Phase 1: 
— Finalizing documentation 


— Ensuring proper monitoring mechanisms are implemented 
Model monitoring 
Data drift 


e Potential Phase 2: 


— Main objective to increase proportion of actioned comments 
Increasing scope to comments with dollar amounts and dates 
Increasing scope fo include more categories (reasons for separation) 
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7. Key Takeaways 


The ROE Comments project is actively removing Comments 
Work Items from Servive Canada queue 

— Over 170k fo date ! 
We are able to implement machine learning solutions 

— in a complex service delivery context 

— |n current IT systems 


Multi-group collaboration works ! 
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